Alignment for Large Engineering Projects: Architecting Distributed Leadership

Presentation on ALIGN process for large engineering project

Smoothing Entailment Graphs with Language Models

The diversity and Zipfian frequency distribution of natural language predicates in corpora leads to sparsity in Entailment Graphs (EGs) built by Open Relation Extraction (ORE). EGs are computationally efficient and explainable models of natural language inference, but as symbolic models, they fail if a novel premise or hypothesis vertex is missing at test-time. We present theory and methodology for overcoming such sparsity in symbolic models. First, we introduce a theory of optimal smoothing of EGs by constructing transitive chains. We then demonstrate an efficient, open-domain, and unsupervised smoothing method using an off-the-shelf Language Model to find approximations of missing premise predicates. This improves recall by 25.1 and 16.3 percentage points on two difficult directional entailment datasets, while raising average precision and maintaining model explainability. Further, in a QA task we show that EG smoothing is most useful for answering questions with lesser supporting text, where missing premise predicates are more costly. Finally, controlled experiments with WordNet confirm our theory and show that hypothesis smoothing is difficult, but possible in principle.Comment: Published at AACL 202

Ternary Through Time: Why Understanding Form Enhances Casual Listening

This presentation will compare the use of ternary form from two different time periods to support our claim that ternary form creates an enjoyable experience for the listener because of the devices used in the form. We will be performing and presenting our analysis of Chopin\u27s Nocturne in C minor (Op. 48 No. 1) and Friends by Joe Hisaishi. The first was written in the mid-nineteenth century and the second in the early twenty-first century, but both maintain a similar structure, showing the effectiveness of this form in creating memorable and gratifying experience. Our research on this topic will benefit those who see it because they will be able to better recognize how the structure of the music they listen to supports their enjoyment of music. Our presentation will also strengthen the idea that while the aesthetic of music may change over time, music across generations has more in common than some listeners may realize and can be enjoyed in similar ways because of the devices employed by these popular forms. Our research will consist largely of analysis of these two pieces, comparing and contrasting the form and devices employed that make these pieces enjoyable, and preparing a performance presentation. We plan to include audience participation to help support our ideas and to connect the audience to the material, and then visually present analysis of the form of these two pieces in conjunction with the performance

Sources of Hallucination by Large Language Models on Inference Tasks

Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as ``indices'' to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.Comment: Findings of EMNLP 202

Sources of Hallucination by Large Language Models on Inference Tasks

Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization, yet this capability is under-explored. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two factors which predict much of their performance, and propose that these are major sources of hallucination in generative LLM. First, the most influential factor is memorization of the training data. We show that models falsely label NLI test samples as entailing when the hypothesis is attested in the training text, regardless of the premise. We further show that named entity IDs are used as "indices" to access the memorized data. Second, we show that LLMs exploit a further corpus-based heuristic using the relative frequencies of words. We show that LLMs score significantly worse on NLI test samples which do not conform to these factors than those which do; we also discuss a tension between the two factors, and a performance trade-off.<br/

Collimation Effects on Magnetically Confined Laser Produced Plasmas

Tokamaks for fusion research are extremely complex and are still limited by inherent instabilities such as material erosion from plasma instabilities. Due to the lack of data and high demand of resources, simulations to portray Tokamaks are essential. A Particle-In-Cell (PIC) simulation for plasma erosion on materials within the Tokamak is to be benchmarked using the experimental data obtained in these experiments. The effects of an axial magnetic field (magnetic field lines are along the plasma propagation direction) on an expanding laser produced plasma plume are investigated. A Continuum Surelite Nd:YAG laser system at 1064 nm wavelength and 6 ns full width half max (FWHM) is used to ablate carbon, aluminum, and boron nitride surfaces in the presence of a magnetic field (~.6T) at 50 mJ, 100 mJ, and 150 mJ under vacuum. The resulting plasma plume is studied using fast photography by employing an intensified charge coupled device (ICCD). The effect of the axial magnetic field changes with the target material. Carbon plume undergoes the creation of side wings that expand perpendicular to the field and curve back into the field after the primary plume has expanded and dissipated. Both aluminum and boron nitride exhibit significant focusing at the center of the magnetic field with no evidence of wings formation. Further work using optical emission spectroscopy is in progress to obtain temperature, electron density, and ionization rate of the laser produced plasma plumes to better understand the mechanism of wing formation as well as plume focusing in different materials